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Abstract 

This paper describes the development and rationale for a human-systems 
integration (HSI) scorecard that can be used in reviews of vehicle specification and 
design. This tool can be used to assess whether specific HSI related criteria have been 
met as part of a project milestone or critical event, such as technical reviews, crew station 
reviews, mockup evaluations, or even review of major plans or processes. Examples of 
HSI related criteria include Human Perfonnance Capabilities, Health Management, 
Human System Interfaces, Anthropometry and Biomechanics, and Natural and Induced 
Environments. The tool is not intended to evaluate requirements compliance and 
verification, but to review how well the human related systems have been considered for 
the specific event and to identify gaps and vulnerabilities from an HSI perspective. 

The scorecard offers common basis, and criteria for discussions among system 
managers, evaluators, and design engineers. Furthennore, the scorecard items highlight 
the main areas of system development that need to be followed during system lifecycle. 
The ratings provide a repeatable quantitative measure to what has been often seen as only 
subjective commentary. Thus, the scorecard is anticipated to be a useful HSI tool to 
communicate review results to the institutional and the project office management. 

Introduction 

Human-Systems Integration (HSI) emphasizes human considerations in systems 
design to optimize the fully integrated system’s (i.e., human and machine’s) perfonnance 
and to reduce lifecycle costs. HSI at NASA is a multidisciplinary field of study composed 
of several user-related areas, including: Human Factors Engineering, System Safety, 
Health Hazards, Manpower & Personnel, Training, and Habitability. 

The incorporation and practice of HSI within NASA is not mandated and 
therefore is not always included in programs and projects most effectively. Development 
of a clearly defined, disciplined, unified, and repeatable HSI process would help ensure 
that the knowledge benefits the appropriate programs/projects. A well-developed HSI 
process would also include standardized HSI tools that support the implementation of the 
process. Proper tools can help capture HSI information from system and sub-system 
managers in a consistent way and they can help track system development. 

Several HSI tools have been developed in the area of military service acquisition 
for manpower, personnel and training integration. Among others, the U.S. Army uses the 
Improved Performance Research Integration Tool (IMPRINT) to describe and analyze task 
networks and to predict the probability of success. Another example, the U.S. Navy’s 
Systems Engineering Analysis Integration Tool (SEAIT) evaluates the effect of 
manpower on ship design, performance and cost (Booher, 2003, Pew & Mavor, 2007). 

The use of HSI tools shortens design time, increase productivity, and lower 
development costs by facilitating human-centered design with early consideration/ 
inclusion of human capabilities and limitations in the design. NASA needed a tool that 
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encompasses all the areas on HSI and can be used at system reviews giving a common 
basis for evaluations and providing an overall picture of the HSI domains. NASA HSI 
team recognized the significance of systematic and quantitative approach to assess the 
level of design maturity and quality of HSI in the spacecraft design and development. 
Thus, the HSI scorecard has been developed for this purpose. 

Background of the HSI Scorecard 

NASA has recently revised the Human Rating Requirements and Guidelines for 
Space Flight Systems (HRR) containing requirements and guidelines for certifying the 
design of all the agency space vehicles carrying humans. In order to facilitate the human 
rating certification, that is, to confirm that the vehicle can fly humans safely, and to 
ensure human-centered design, the Constellation Program also developed the Human 
Systems Integration Requirements (HSIR) that drives the design of space vehicles, their 
systems and equipment with which humans interface in the Constellation vehicles like the 
Orion Crew Exploration Vehicle. These requirements ensure that the design of 
Constellation vehicles is centered on the needs, capabilities and limitations of the human. 

The HSI scorecard under development is loosely based on the HRR and HSIR. It 
contains broad areas such as HSI Process, Human Systems Interfaces, and Human 
Performance Capabilities, among others. Each general area contains one or more sub 
categories, such as design team composition under HSI Process. All these are rated based 
on how well expectations (1 - unsatisfactory to 5 - exceptional) were met at a particular 
event, such as a technical review of a specific feature, or an integrated cockpit layout 
evaluation. This type of rating allows for assigning a score in each area, as well as 
obtaining an overall score for a system. 

The HSI scorecard is intended to be used at technical reviews of specification 
documentation and design, preliminary design review, detailed design review and other 
reviews that are conducted by a system engineer, a system manager or a human factors 
engineer concerned with human health and performance. The scorecard can be broken up 
into sections and the individual sections can be used by subject matter experts. For 
example, environmental factors can be evaluated by a toxicologist or a lighting specialist. 
Similarly, the human systems interface rating may be conducted by a human factors 
engineer. 

Beyond designing a system according to requirements, there is also a need to 
evaluate a system based on how well the HSI process has been followed and how well 
HSI guidelines have been implemented. The system can also be evaluated more in-depth, 
such as whether or not sensory capabilities of humans were considered in the design 
process. 

Overall, the HSI scorecard can provide a more complete picture about the 
progress of system design at different phases of the lifecycle. This will be a useful HSI 
tool for systems engineers as well as to human factors engineers and other specialists to 
communicate the acceptability of the HSI aspect of the design in support of human rating 
and certification. 

Motivation for the HSI scorecard 

The HSI scorecard is a tool for assessing whether or not specific HSI related 
criteria have been met as part of a project milestone, event or product. The intent of the 
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tool is not to look line by line at whether or not requirements have been used at the right 
level, and whether they have been verified, but to review how well the human as a system 
has been considered for the specific event/product and to identify major gaps and 
vulnerabilities from an HSI perspective at different design and development phases. 
Although the review may be subjective, this scorecard attempts to provide rigor, or 
quantitative measure to the subjective commentary. 

The scorecard tool is intended to provide a high level snapshot that can be 
presented by an HSI lead to the Health and Medical Technical Authority (HMTA) and 
Institutional Management of Space Life Sciences Directorate (SLSD) for concurrence as 
well as to the Project Management team to give an overall summary of the state of the 
human system requirements incorporation for a particular event/product. In addition, this 
data will help independent assessment of the vehicle by the SLSD team to obtain 
SLSD/HMTA approval of the Constellation vehicle. The results of the assessments 
would capture evidence for human rating the vehicle and provide Program Office 
personnel the issues or vulnerabilities from a human system perspective. 

Detailed description of the HSI Scorecard 

During the lifecycle of the system several reviews are conducted at different 
stages to assess the design and development process. Such events could include system 
baseline review, preliminary design review, critical design review, technical reviews, 
crew station reviews, mockup evaluations, table top reviews, cockpit working group 
assessments, and review of major plans or processes. The products from these reviews 
include system/subsystem specifications or data books, analysis reports, and engineering 
drawings. However, in general, these reviews do not necessarily provide a full picture of 
how and whether the HSI process has been implemented during the phases. 

The items in the scorecard are rated on a scale from 1 to 5 based on whether they 
meet the expectations for the current phase of the system (e.g., pre-PDR, PDR) or not. If 
a system is expected to have a 10% maturity level being at the beginning of the design 
phase, it can still be rated 5 (exceptional) on the rating scale if all expectations have been 
exceeded for that maturity level. 

The rating scales also provide the option not to rate an item in case it is not 
relevant to the review or system. Furthennore, each item has a section for comments to 
identify major pros/cons and vulnerabilities for that item. General comments may be 
added at the end of the form. The scorecard also tracks system and subsystem name, type 
of review, maturity expectation, and design phase along with reviewer information and 
date. 

Areas in the HSI scorecard 

Based on the topics covered in HSIR, there are 10 areas included in the scorecard: 
HSI process, human system interfaces, anthropometry and biomechanics, human 
perfonnance capabilities, natural and induced environments, crew safety, health 
management, architecture, hardware and equipment, infonnation management. 

The HSI process area refers to the involvement of human system, human design 
and HSI human factors personnel in the design process. The human interfaces area has 
items referring to how well interfaces are designed with the human as a system in mind, 
including use of task analysis, human centered design practices, and human-in-the-loop 
testing. There are items on anthropometries and biomechanics focusing on how well 
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human physical characteristics are considered in the design, such as strength and 
accessibility, and reach analysis and crew population/size. 

In the human performance capabilities section the items are concerned with 
considerations of human performance and sensory capabilities, such as gravity (G) 
condition, and physical/mental workload in cases where human response is required as 
system input. Items referring to considerations of atmospheric conditions, acoustics, 
temperature, vibration, radiation, and lighting are in the natural and induced 
environments section. The health management area refers to the human system design 
considerations to include management of crew health in areas such as nominal/ 
contingency landing, crew survivability, nourishment, exercise, health monitoring, 
communication, and rest. Crew safety has a section as well as system architecture that is 
addressed in terms of habitable volume analysis, ability to perform required tasks. 

Finally, there is a section on hardware usability and maintainability as well as 
information management. 

Development of the HSI Scorecard 

The first version of the HSI scorecard was developed based on the main 
categories that are used to organize information in HSIR. Items were added based on 
importance and generality with summarizing requirement language. Based on discussions 
with HSI and systems engineers, the scale used from 0 to 100 in 25% increments to 
indicate to what percent expectations have been met. After several iterations, the scale 
was changed to a simple Likert scale to make it easier to understand for raters. The scale 
goes from 1 - unsatisfactory to 5 - exceptional (see Figure 1). 

The scorecard has been used in several reviews for Orion, Crew Exploration 
Vehicle (CEV) and it was revised based on comments received from these reviews. The 
description of the items has been changed to make it more appropriate for the users. Items 
were added or removed depending on their relevance. For example, all HSIR topics 
related to extravehicular activity (EVA) and ground operations were removed. These may 
be added back to an EVA centered version of the scorecard. 


Please insert Figure 1 about here 


Moreover, we conducted in-depth interviews with users to get feedback on the 
scorecard. Most reviewers used the scorecard three or more times at technical reviews. In 
general, the electronic version was preferred to the paper version; however, the paper 
version was also deemed necessary to provide flexibility when using a computer is not 
feasible. Users think the scorecard provided a broader picture for the review and a high 
level overview that otherwise would not be available. In addition, they reported that the 
tool helped them to be consistent in their assessments of different systems/sub-systems. 

Reviewers reported that filling out the scorecard takes a fair amount of time. This 
may not be a disadvantage from the point of view of the HSI lead because it makes sure 
that the review thorough, especially since this data will help to determine if the vehicle 
design is human-rated for a space flight. 
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Reporting with the HSI Scorecard 

One purpose of the scorecard is to provide a concise report to program managers 
about the development status of a system. Depending of the details needed, scores or 
scores averaged for each area along with comments may be reported. For one review, the 
result may be summarized in a figure (see Figure 2) showing the mean ratings for each 
area. This kind of visualization helps with finding at a glace the areas that perform below 
expectation and areas that are doing as expected. 


Please insert Figure 2 about here 


For a series of reviews the results may be shown as a figure and or table (Figure 3 
and Table 1). Summarizing several reviews in one figure or table provides a picture of 
system development process in time. Areas that have changed dramatically over time can 
be easily pointed out; these may be the areas that need more attention in the next 
lifecycle. 


Please insert Table 1 about here 


Please insert Figure 3 about here 


Benefits of the HSI Scorecard 

During the development process and evaluations, the scorecard proved to be beneficial in 
several respects, beyond providing consistent ratings of systems. 

a) Negotiation tool . The scorecard offers common language, basis, and criteria for 
discussions among system managers, evaluators, and design engineers. Due to this, the 
scorecard can be used as a negotiation tool in reviews and help early/critical inclusion of 
the project prime contractor’s human factors/ HSI personnel in the design and 
development efforts and their products. 

b) Look-ahead tool . The scorecard items highlight the main areas of system development 
that need to be followed during system lifecycle. The scorecard specifies common issues 
on a high level that helps focus on all areas of development. 

c) Repeatability . The ratings, even though they are subjective, provide a repeatable 
quantitative measure. As reviewers get more familiar with the scorecard, the ratings will 
be easier to complete and inter-rater differences will also decrease. 

d) Traceability in time. Ratings can be tracked and plotted as time-series data. This will 
allow following the development of a system in time based on the same criteria. 
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Furthermore, it also makes easier to identify areas of improvement for new development 
processes. 

e) Marketing tool for HSI. The scorecard can be a good marketing tool for HSI providing 
the general audience more familiarity with the process. 

Conclusions 

Overall, preliminary comments on the scorecard after being used during the CEV 
technical reviews demonstrated that such HSI tools were instrumental in a systematic and 
structured assessment of the vehicle design. HSI tools can help track changes in systems 
and if used consistently at important phases of the lifecycle, they can provide infonnation 
for cost-benefit estimations as well. Future work on the HSI scorecard will focus on using 
the tool for cost-benefit evaluations of early inclusion of human as a system within the 
vehicle design. Finally, different reporting formats will be developed to document the 
summary results for up and out communication. 
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Figure 1. Screenshot of the first page of the scorecard. The reviewer can rate an item or 
specify reasons for not rating an item. There is a “Comments” section as well to allow 
adding more details. 
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Figure 2. A possible visualization of average hypothetical ratings of a system on a 
review. 


Review 1 


Rating (scale 1 to 5) 
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Figure 3. Graph showing hypothetical results from five reviews for the same system that 
were conducted in different phases of the lifecycle. This allows for an overview of 
system development. 
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Table 1 

Hypothetical ratings of a system across all areas in five reviews. 



Areas 

Review 1 

Review 2 

Review 3 

Review 4 

Review 5 

1 

Human-Systems Integration Process 

2.70 

1.60 

3.75 

1.70 

3.30 

2 

Human-System Interfaces 

3.35 

2.25 

3.20 

2.80 

3.90 

3 

Anthropometry and Biomechanics 

3.95 

4.00 

4.30 

2.35 

1.65 

4 

Human Performance Capabilities 

1.25 

2.70 

1.65 

2.80 

2.85 

5 

Natural and Induced Environments 

3.85 

2.75 

3.75 

4.30 

3.25 

6 

Crew Safety 

4.45 

3.90 

4.85 

1.75 

4.35 

7 

Health Management 

4.00 

3.35 

2.25 

3.80 

3.65 

8 

Architecture 

4.35 

3.45 

1.50 

3.45 

2.85 

9 

Hardware and Equipment 

3.25 

3.10 

2.70 

3.90 

4.30 

10 

Information Management 

2.80 

3.65 

3.35 

2.80 

2.80 
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